COCOMO II Calibration and Plans

COCOMO II is an effort to update software cost estimation models, such as the 1981 COnstructive Cost MOdel and its 1987 Ada COCOMO update. Both these models have experienced difficulties in estimating software projects of the 90s due to challenges such as non-sequential and rapid-development process models; reuse-driven approaches involving commercial-off-the-shelf (COTS) packages, reengineering, applications composition, and application generation capabilities; object-oriented approaches supported by distributed middleware; software process maturity effects and process-driven quality estimation. The COCOMO II research effort aims at alleviating these problems and is concentrated on developing a model well-suited for the 1990s and then annually updating it for the forthcoming years of the 21 Century. The initial definition of the COCOMO II model and its rationale are described in [Boehm et al 1995]. The model has three sub-models, Applications Composition; Early Design; Post-Architecture, which target the realm of future software practices marketplace. The Applications Composition model is used primarily on projects incorporating Integrated Computer Aided Software Engineering Tools. The projects are typically small and are developed by relatively small teams in a few weeks or months. The Applications Composition model is based on Object Points[Banker et al 1994, Kumar et al 1993]; that are a count of the screens, reports and 3GL modules, each weighted by a simple/medium/difficult complexity factor. The models objectives are to resolve potential high-risk issues such as complex UIs, software/system interaction etc. by early prototyping efforts. Object points are a good input to the Applications Composition Model as they are match the degree of information known for such development efforts. The Early Design Model is used to obtain a rough estimate of the software under development. It is employed when fine-grain information is not available to support detailed cost estimation. The Early Design model is extremely beneficial in exploring alternate software/systems architectures. It uses Function Points and a small set of cost drivers as inputs. The Post-Architecture Model, as the name suggests, is typically used after the software architecture is well defined and established. It estimates for the entire development life cycle of the software product and is a detailed extension of the Early-Design model. This model is the closest in structure and formulation to the Intermediate COCOMO ’81 and Ada COCOMO models. It uses Source Lines of Code and/ or Function Points for the sizing parameter, adjusted for reuse and breakage; a set of 17 effort multipliers and a set of 5 scale factors, that determine the economies/diseconomies of scale of the software under development. The 5 scale factors replace the development modes in the COCOMO ’81 model and refine the exponent in the Ada COCOMO model. The Post-Architecture Model has been calibrated to a database of 83 projects collected from Commercial, Aerospace, Government and FFRDC organizations. This paper describes in detail the calibration process incorporated and summarizes the results obtained. The Early Design Model calibration is obtained by aggregating the calibrated Effort Multipliers of the Post-Architecture Model as described in[USC-CSE 1997a]. The Scale Factor calibration is the same in both the models. The Applications Composition Model has not yet been calibrated due to limitations in the availability of project data. The Post-Architecture Model calibration process began by assigning expert-determined values to the 22 (17 Effort Multipliers + 5 Scale Factors) parameters. These assigned values were iterated with USC-CSE’s affiliates to obtain the initial version of the model, now known as the “A-Priori Post-Architecture Model”. Once the model was well-defined; an extensive data collection activity was initiated. This activity resulted in an average sized MS Access database of 83 projects with complete ready-to-analyze data. Statistical analyses exhibited high correlation among four of the 17 Effort Multipliers which led to consolidating them into 15 Effort Multipliers (discussed further in the main paper). The data was then divided into two subsets; one subset of 59 datapoints that was used for the regression analysis and the other subset of 24 projects that was used for cross validation. Multiple Regression Analyses on the first subset of 59 datapoints determined the coefficients for the 20 (now 15 Effort Multipliers + 5 Scale Factors) parameters. A Bayesian approach was used to adjust the A-Priori expert-determined model parameters to obtain the calibrated “A-Posteriori Post Architecture Model” that reflected the characteristics of the actual 83 projects data. The Bayesian technique was preferred over a pure least squares approach due to uncertainties that were apparent in our subjective data. Once, the A-Priori Post Architecture Model was calibrated; it was cross-validated using the second subset of 24 datapoints. Prediction Accuracy was computed in terms of Proportional Error (PE) which had a Normal Distribution and the following accuracies for Effort Prediction were observed : Prediction Before Stratification by Organization After Stratification by Organization PRED(.20) 46% 49% PRED(.25) 49% 55% PRED(.30) 52% 64% In the above table, the column “Before Stratification by Organization” represents the Prediction Accuracy obtained by using the “A-Posteriori Post Architecture Model” on the 83 datapoints before any data stratification. On the other hand, the column “After Stratification by Organization” represents the Prediction Accuracy obtained by using the “A-Posteriori Post Architecture Model” on the 83 datapoints after stratifying the data by organization and computing a new multiplicative constant for each of the organizations. It is clear from the above table that simple local calibration helps in improving the prediction accuracy of the model. The COCOMO II research team hopes to update the now published COCOMO II.1997 version by continuously collecting data from its affiliates and other sources and calibrating the model to the enhanced database. By progressively giving more weight to the data-determined model parameters and less weight to the A-Priori model parameter values; the team’s aim is to develop a more statistically sound model. Other emerging extensions to the COCOMO II model include Cost/Quality modeling, COTS modeling and Estimation based on Activity Distribution. * Note : Stratification “by Organization” does not mean “by Application Type”. In the COCOMO II database, we have actual project data from seven organizations and the data was stratified into seven sets based on the source of the data.